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1 Introduction: 


This final report documents the activities performed on NASA Grant #NCC 2-967 during the 
period 1 April 1996 through 30 September 1997. 


2 Attachments: 

The work performed in this program and relevant background material are documented in the three 
papers which are attached to this document. 
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ABSTRACT 

The Stanford University Aerospace Robotics Lab 
(ARL) has developed a fully autonomous hoverable 
unmanned air vehicle (HUAV) which uses differen- 
tial carrier phase GPS (DCPGPS) as the only sen- 
sor for both control and navigation. Precision au- 
tonomous operation in a structured environment has 
been demonstrated at the 1995 Aerial Robotics Com- 
petition, where the ARL helicopter demonstrated re- 
trieval and transportation of 4 oz objects around a 
small field. 


The ARL has experimentally demonstrated the advan- 
tages of combining differential carrier phase GPS for 
high-bandwidth vehicle control with computer vision 
to enable the operation of an unmanned, autonomous 
helicopter in a dynamic unstructured environment. 
DCPGPS techniques and conventional control meth- 
ods are used to close a short-period inner loop, stabi- 
lizing the helicopter’s dynamics and providing a global 
navigation system. An outer loop is closed using feed- 
back from the vision system to determine the precise 
location of the ground and other objects in the envi- 
ronment. 

This paper examines the combination of differential 
carrier phase GPS and computer vision techniques to 
enable an autonomous vehicle to operate in unstruc- 
tured environments. Experimental results are pre- 
sented and future work is discussed. 

INTRODUCTION 

Helicopters have found broad use in many diverse 
applications, including search and rescue, fire fight- 
ing, fine-scale terrain mapping, and agricultural oper- 
ations. The operation costs of manned helicopters are 
quite high. It is expected that the operational use of 
unmanned helicopters could greatly reduce the cost of 
many of these tasks. 

The remote piloting of hoverable unmanned air vehi- 
cles is a very difficult task, requiring great operator 
skill and attention as well as a high-bandwidth, low- 
delay data link between the pilot and vehicle. Further- 
more, most applications require that the pilot maintain 
visual contact with the HUAV at all times. These re- 
quirements limit the applications of HUAVs and pro- 
hibit large scale deployment of these systems. 

Many airborne applications require operation in dy- 
namic, unstructured environments. Imaging is the 



most natural method of obtaining information about 
an unknown environment, and has broad applica- 
tions in military reconnaissance, agricultural inspec- 
tion, and mapping. Imaging plays a critical role in the 
operation of HUAVs, providing information to ground 
based operators and enabling operation in an unstruc- 
tured, dynamic environment. 

The Stanford Aerospace Robotics Laboratory has been 
working towards developing prototype HUAV systems 
which address these issues. This paper examines the 
combination of differential carrier phase GPS and com- 
puter vision techniques to enable an autonomous ve- 
hicle to operate in unstructured environments. 

EXPERIMENTAL SYSTEM 

The HUMMINGBIRD autonomous helicopter was 
originally constructed for research in the control of an 
unstable flying machine using Global Positioning Sys- 
tem measurements as the only sensor. The hardware 
system, shown in Figure 1, has now been extended to 
deal with the combination of GPS measurements with 
vision system measurements. The basic airframe is a 



Figure 1: Current configuration of the Stanford 
HUMMINGBIRD. 

heavily modified Schluter Futura hobbyist model he- 
licopter. The airframe is capable of carrying twenty- 
five pounds of experimental equipment. Nominal gross 
takeoff weight is forty-six pounds. 

A plywood plate, clamped to the bottom of the he- 
licopter landing gear, supports the onboard electron- 
ics and their power supplies. Four GPS antennae are 
mounted on the helicopter — the port and starboard 
antennae are mounted on wooden towers at the ex- 
treme aft corners of the plywood plate, the nose an- 
tenna is supported by small aluminum tubes, and 
the tail antenna is mounted directly onto the heli- 
copter tail boom. The electronics consist of a 486- 


based computer, a wireless ethernet modem, an RS232 
radio modem receiver, two GPS receivers (Trimble 
TANS- Vectors), two color cameras, and one microwave 
wireless video transmitter. All electronics are shock 
mounted on closed-cell foam mounts. 

Critical to the safe flight testing of the helicopter is a 
fully manual flight mode. An independently powered 
radio receiver, with computer override capability, can 
be used to take manual control of the helicopter at 
any time. Flight testing is always performed under 
the direct supervision of a skilled RC pilot. 

Ground station equipment consists of a dual-processor 
Pentium computer (coupled to the wireless ethernet 
network), a single antenna GPS receiver, an RS232 
radio modem transmitter, and a microwave wireless 
video receiver. The GPS receiver on the ground re- 
ceives signals from up to six satellites. Carrier phase 
measurements are uplinked to the helicopter directly 
through the single direction dedicated RS232 link. No 
GPS processing is performed on the ground. 

The helicopter receives GPS signals from three 
sources — the ground station (via the RS232 link), from 
a local “position” GPS receiver which produces raw 
carrier phase signals from one antenna, and from a lo- 
cal “attitude” GPS receiver which produces differential 
measurements between a master antenna and each of 
three slave antennae. The helicopter’s on board 486 
then applies algorithms as described in Conway [1] 
to determine the vehicle attitude, attitude rate, po- 
sition, and velocity. No real-time computation using 
the video information is currently done onboard the 
helicopter. 

Before the helicopter begins flight, the planned trajec- 
tory is determined and uploaded to the onboard com- 
puter. During flight, the ground station computer re- 
ceives and stores video images marked with the corre- 
sponding GPS-determined helicopter position and at- 
titude. After landing, the ground station downloads 
all data taken by the onboard computer during flight 
and begins a post-flight analysis. The video data is 
then tested for validity and the final processing is per- 
formed. 

The wireless ethernet communciation between the he- 
licopter and the ground-based Pentium system allows 
the helicopter to utilize these off- vehicle computer re- 
sources. In the near future, it is envisioned that real- 
time vision algorithms could be applied to direct the 
control of the helicopter at a high level. 


DIFFERENTIAL CARRIER PHASE GPS 

Key to the success of the HUMMINGBIRD platform 
has been the use of GPS as a sensing technology for 
closed loop control of an unstable air vehicle. GPS 
attitude and position sensing has several distinct ad- 
vantages, including: 


appropriate since GPS provides an excellent sensor for 
position, while vision provides a large amount of local 
information about the surrounding environment but 
provides little information about the global position 
of various objects. 

GPS-AUGMENTED VISION 


• no moving parts 

• very little calibration required 

• drift-free in both position and attitude 

• future promise of reduced size, power consump- 
tion and increased reliability associated with elec- 
tronic parts. 


A major achievement of this project is the fully au- 
tonomous flight of a helicopter using GPS as the only 
sensor for both attitude and position stabilization and 
control. A 10 Hz update rate for all 6 DOF is suf- 
ficient to ensure stable flight of our test vehicle. No 
other rate gyros, compasses, et cetera were used. For 
a complete description of the algorithms used in this 
work see Conway [1]. 

The ability to fly pre-programmed paths was demon- 
strated in the 1995 International Aerial Robotics Com- 
petition, where our helicopter was the first — and to 
date only — air vehicle to pick up and move a small 
ferromagnetic disk fully autonomously. The trajectory 
flown is shown in Figure 2. 



Figure 2: Competition flight trajectory 

The helicopter autonomously retrieved a disk in 
the Pickup Ring , crossed the 1-m barrier at an 
altitude of six meters , and delivered the disk to 
the Drop-off Ring. 


As previously stated, DCPGPS techniques and con- 
ventional control methods are used to close an inner 
loop, stabilizing the helicopter s dynamics and provid- 
ing a global navigation system. An outer loop is closed 
using feedback from the vision system to determine the 
precise location of the ground and other objects in the 
environment. 

The precision available using DCPGPS is useful when 
it is coupled with traditional computer vision tech- 
niques. Many such techniques exist for localizing ob- 
jects within a scene relative to the camera. If one 
can directly measure the position and orientation of a 
camera, one can then determine the location of sensed 
objects in global coordinates. 

In the first set of experiments performed using the 
HUMMINGBIRD helicopter, the UAV was maneu- 
vered over a grass field upon which many distinctive 
objects were placed. Assuming the field is level, and 
given the camera’s height above the ground (measured 
with GPS), a single image is sufficient to determine the 
location of a target in global coordinates. 

It is possible to extend this technique to provide a 
crude “mosaicking” capability by capturing successive 
images, each with corresponding DCPGPS position 
and attitude. By appropriate image transformations, 
a large image can be constructed from many, possibly 
overlapping, images. 


3 2 1 

Target Locations 

4 5 6 


O 

T akeoff 
Point 


Figure 3: Test Field 


Further work has begun to extend the GPS sensing 
system to become an integral part of a computer vi- 
sion system. The mixing of the two technologies seems 


Figure 3 shows a typical test setup for the object lo- 
cation algorithm. The test field measures 15 meters 
by 9 meters, with a known take-off point five meters 




outside the surveyed area. Targets, which were black 
plastic barrels or 4-inch metal disks painted orange, 
were placed at presurveyed target locations within the 
field. 
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Figure 4: GPS- Augmented Vision - Experimen- 
tal Results 

Figure 4 presents the results from three test flights 
over the field. In each test, objects were placed on two 
of the six possible target locations, and HUMMING- 
BIRD then proceeded to survey the disk locations from 
an altitude of about three meters. Each data point 
corresponds to a single photograph of the field. 


Table 1: Vision Based Object Location 


Test 

Target 

Position Error (m) 

Number 

Number 



1 

5 

0.27 

0.39 

1 

6 

0.31 

0.52 

2 

1 

0.21 


2 

4 

0.46 
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As Table 1 shows, our accuracy at locating the disks 
is quite good. The mean location is generally within 
0.50 meters, and the standard deviation usually un- 
der 15 centimeters. The data clearly indicates that 
the greatest error sources are constant offsets, which 
will be eliminated in the future with better algorithms 
and improved equipment calibration. Closer synchro- 
nization between the GPS attitude solutions and the 
camera images should also help to reduce the small 
amount of data scatter present. 


FUTURE RESEARCH 

The ARL is currently working on several additional 
projects to further implement vision with the HUM- 
MINGBIRD helicopter. These include stereo imaging, 
motion based stereo, servoing off of camera inputs, and 
mobile object tracking and following. 

In many computer vision applications, target objects 
are not located at known distances from the camera. 
In these situations a single image is not sufficient to de- 
termine the location of the object. HUMMINGBIRD 
employs a pair of stereo cameras to enable it to de- 
termine the range to target objects. This will help 
facilitate the retrieval of objects from on top of obsta- 
cles, or when the altitude of the target object is not 
known beforehand. 

Conventional stereo vision is only effective out to a 
limited range determined by the baseline distance be- 
tween cameras. Because HUMMINGBIRD is capable 
of determining the locations and pointing angles of its 
onboard cameras quite accurately using GPS, it is able 
to perform motion based, or long-baseline, stereo. This 
involves taking two images from different locations and 
then triangulating to determine the global location of 
the object. Eventually, this could provide the capa- 
bility to perform accurate stereo mapping of objects 
kilometers away. 

In the future, HUMMINGBIRD will be able to servo 
off of its vision system by working to keep a target 
object centered in its field of view and at a constant 
range. This will greatly facilitate object retrieval or 
manipulation and allow for greater resolution when 
performing aerial surveying tasks. HUMMINGBIRD 
will also be able to track and follow moving objects, 
improving its ability to work in dynamic environments. 
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1 Introduction 

This paper describes the Stanford Aerospace Robotics Laboratory’s (ARL) entry in the 
1997 Association for Unmanned Vehicle Systems’ 1997 Aerial Robotics Competition. The 
objective of the competition is to demonstrate a fully autonomous air vehicle which can: 

• Take off from a 15’ square at a given location. 

• Overfly a 120’ by 60’ grass field without crossing its boundaries. 

• Determine the location of five 55-gallon plastic barrels. 

• Identify the type of barrel as either biohazard, radioactive, or picric acid.. 

• Find and retrieve a ferromagnetic disk, located on one of the barrels. 

• Return to the take off location and land. 

A complete description of the competition and official rules can be found in [5] . 

2 The Stanford Aerospace Robotics Laboratory 

The ARL has been working for many years on technologies relevant to the Aerial Robotics 
Competition. These technologies include 

• “Object-Based Task-Level Control” - allowing the user of an automated system to 
command a task, rather than be consumed with the low-level control issues of the 
task. 
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• GPS for real-time control. 

• Vision for real-time control. 

• Path planning in structured and semi-structured environments. 

• Real-time “on-line” system identification. 

The Stanford Department of Aeronautics and Astronautics has several other research 
laboratories. Relevant to the competition are the the Aircraft and Flight Research Labora- 
tory, and the Global Positioning System (GPS) Laboratory. The members of the Aircraft 
and Flight Research Laboratory have decades of experience in design and construction of 
free-flying model aircraft, and their application to flight research. The Global Positioning 
System Laboratory at Stanford is the Federal Aviation Administration’s center of excellence 
for GPS technology. 

3 Major System Components 

3.1 Aircraft Selection 

In order to expedite the development of this robotic system, it was decided to utilize an 
“off the shelf” air vehicle, as opposed to designing and constructing one in-house. The first, 
second, and fifth competition objectives listed above eliminated the use of any fixed-wing 
aircraft. A Schluter Futura model helicopter was selected for its low empty weight, its large 
payload capacity (over 25 pounds), and its 15 minute endurance. This Futura was extensively 
modified, particularly its powerplant. A two-horsepower single-cylinder engine was replaced 
by a five-horsepower dual-cylinder engine intended for use on model airplanes. This required 
the design and construction of a new engine mounting plate and modification of the power 
transmission devices. 

The Futura is equipped with two stabilization devices, “Hiller paddles” and a mechanical 
rate gyro. The “Hiller paddles” are used to slow the lateral and longitudinal dynamics. The 
rate gyro is used in a simple electrical feedback loop to slow the yaw dynamics. Both of 
these devices are standard equipment for helicopter of this size, and are essential to permit 
manual flight operation. 

3.2 Sensor Selection 

Sensors that are able to measure attitude are necessary to stabilize the helicopter system. 
This was confirmed with flight tests, where an unsuccessful attempt was made to stabilize 
the helicopter without using attitude information. In order to navigate, a three degree of 
freedom position sensor is required. 

Inertial navigation systems were ruled out due to high mass and high cost. Combinations 
of magnetic compasses, rate gyros (for stabilization), and ultrasonic sensors were eliminated 
due to weight constraints. 

The competition rules specify that the extent of the playing field and the contestant’s 
position on the field can be determined a priori. Thus a navigation system which gives 
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only relative position (with respect to a known position on Earth) is sufficient to complete 
the competition objectives. One such system is the Global Positioning System (GPS). This 
system has several major advantages, including: 

• High azimuth angles of the broadcast signals. This avoids occlusion often present in 
terrestrially based broadcast navigation systems. 

• Availability anywhere in the world. 

• Integration of all sensors into a single unit. 

• Rate information “at no extra charge” . 

• No moving parts. 

• Measurements with respect to the earth fixed reference frame. 

• Relatively small size and power consumption. 

The use of GPS as the only sensor for the control of an unstable vehicle poses a significant 
challenge and advancement in the development of GPS as a sensing technology. 

GPS was selected as primary sensor for the stabilization of the Stanford ARL’s au- 
tonomous helicopter. A magnetic pulse counter was also added to measure engine RPM to 
be used for better control of the collective/throttle altitude subsystem. Due to the unstruc- 
tured environment of the competition, however, sensing of the external situation around the 
helicopter was necessary for intelligent navigation. Two on-board video cameras were added 
to allow stereo calculations and computer vision processing of the helicopter’s environment. 
Wireless video transmitters were mounted on the helicopter so that live information could 
be sent to a more powerful ground station for processing. Using a modified version of a 
Teleos vision [6] system, stereo ranging, object motion tracking, and color image processing 
techniques are possible. 

4 System Design 

The primary design consideration during the development of the ARL helicopter was to 
minimize risk of crashing throughout the program testing. The helicopter’s controls are 
configured to permit easy human intervention, in order to deal with unexpected malfunctions. 

The helicopter receives GPS signals utilizing four independent antennas. All four of 
these signals are demodulated by a single GPS receiver, which produces all the information 
necessary to determine vehicle attitude, and attitude rates. One of the antennas is also 
fed into a second GPS receiver, which determines approximately half of the information 
necessary to determine vehicle position and velocity. 

On the ground, a fifth (stationary with respect to the earth) GPS antenna receives signals 
similar to those received in the air. A ground based GPS receiver demodulates this signal, 
obtaining the second half of the information required to determine vehicle position. 

An on-board 486 computer receives all information from the three GPS receivers via serial 
communication links. The serial link between the helicopter and the ground is made via a 
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two-way radio link. The 486 computer completes the calculation of the vehicle position, 
velocity, attitude, and attitude rate, and then determines an appropriate control output. 
These outputs are then fed through the manual control system to the helicopter’s servos. 

4.1 Reliability Considerations 

In order to permit rapid human intervention, all commands sent to the helicopter’s control 
servos pass through a reliable, independently- powered manual control system. A human 
pilot can override automatic operation in one of two ways: 

• Toggling a switch on the control panel returns the helicopter to complete manual 
control. 

• Disturbing the controls causes the manual control inputs to be algebraically summed 
to the autonomous control inputs. 

A design objective of the ARL helicopter was to permit frequent and rapid modifications 
to the computational algorithms, without adding significant risks to the survivability of 
the helicopter. It was decided to utilize a timer card to handle the interface between the 
manual controller and the 486 computer. The code on this card was developed and carefully 
tested prior to its use in flight, and has not been modified since this rigorous verification. 
One QUARTZ-MM timer card, attached to the 486 computer, is used in this interface. It 
converts pulse-width modulation of eight standard model aircraft channels to digital signals 
and vice versa. 

In order to ensure a reliable power source for the most vital systems, an independent 
battery is used to power the fuel shut-off servo, the manual controller, and an independent 
ground-controllable multiplexer. A second battery powers the helicopter’s servos. A third 
battery group consists of two subgroups of 6 and 10 cells, which power the 486 computer, 
a wireless Ethernet modem, the GPS receiver, cameras and video modems, and all other 
equipment associated with automatic control. The second battery was added in order to 
decouple electrical noise from the manual controller, to increase endurance, and to decouple 
the (critical) servo system from the often modified automatic control electronics. The third 
group was split to provide unregulated 7.2 and 12 volts to the electrical components. 

In compliance with Air Vehicle Attribute 3, a ninth channel of the manual controller is 
connected directly to the fuel shut off servo. The ground-controllable multiplexer combines 
the signals produced by the computer with those sent manually, including the signal to shut 
off the fuel, to present a consistent input to the helicopter servos. Should any system become 
inoperative, rendering the helicopter a hazard, the ninth channel will allow the helicopter’s 
engine to be shut down. 

4.2 GPS Antennas 

Maintaining GPS signal integrity during flight operations is critical for successful flight op- 
erations. Issues addressed in design of the antenna system were multipath interference, the 
relative geometry of the antennas, and satellite visibility. 


4 



One of the major drawbacks of GPS signals is the line of sight propagation characteristics. 
In order for a signal to be received by a GPS antenna, an unoccluded line of sight (roughly 
speaking) must exist between each antenna and the various satellites broadcasting relevant 
signals. 

The initial GPS attitude determination algorithm invented by Cohen [2] required that the 
four antennas be mounted in a non-coplanar fashion, and that the antennas have unoccluded 
lines of sight to the identical constellation of satellites. In an aircraft application, these 
conditions are maintained by by mounting the antennas high on top of the fuselage, wings 
and tail. 

With a helicopter, it is difficult to mount the antennas above the main rotor. Work by 
Conway [3] extended attitude algorithms to include coplanar configurations, and configura- 
tions where some satellites are occluded when viewed from individual antenna. 

As a consequence of this extension, the ARL helicopter is able to mount the four antennas 
nearly in a plane, just below the main rotor disk. This provides nearly clear view of the sky 
by all antennas, with only a small occlusion by the main rotor mast. Since the rotor blades 
are constructed of wood, have a small cross section (6 cm) when compared to the wavelength 
being considered (19.2 cm), and are spinning rapidly, there is little interference as a result 
of the antennas being below the rotor disk. The small amounts of interference induced are 
eliminated by the phase lock loops in the GPS receivers. One antenna (the master antenna) 
is mounted on the tail boom, one is mounted on the fuselage forward of the main rotor mast, 
and two antennae are mounted to the plate holding the electronics. 

4.3 GPS Receivers 

At the time of selection, few receivers were available which were capable of computing air- 
craft attitude as well as position, with relatively low total mass. The Trimble TANS Quadrex 
was selected as the best candidate in this regard. In order to maintain compatibility with 
this system, the Trimble 16248-50 antennas were selected. These antennas have crystal RF 
filters which provide good frequency domain side-lobe attenuation, while providing approxi- 
mately 50 dB gain from built in amplifiers. These antennas are somewhat heavier than other 
available antennas; however, the superior in-band gain and out-of-band attenuation of these 
antennas justified the weight penalty. 

An RF splitter allows the tail boom’s antenna (the master antenna) to be distributed to 
both the attitude GPS receiver and the position GPS receiver. The receivers are electrically 
identical but have significantly different software operating on their local microprocessors. 

The interface between each GPS receiver and the 486 computer is made through 38400 
baud serial communication links. Configuration commands are sent to the GPS receivers 
upon system start up, after which position and attitude information is made available to the 
486 computer ten times per second. Information from the ground station is also received by 
serial link, in this case a 9600 baud, wireless 461 MHz modem with an RS-232 interface. 

4.4 Main Computer 

Some processing is performed by the 486 computer in order to resolve the GPS information 
into earth fixed coordinates (for position), and locally level (for attitude). These calculations 
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result in vehicle position, attitude, velocity and attitude rate ten times per second. Position 
is accurate (RMS values) to approximately 3 centimeters in all three axes (depending on 
satellite geometry), and attitude to about one degree (which is a function of antenna ge- 
ometry). The velocity is accurate to about 10 centimeters per second, and attitude rate to 
about 1 degree per second. 

Once the 486 computer has calculated both position and attitude, an appropriate control 
signal is determined and sent to the timer card for conversion to pulse-width modulation 
signals. These signals are combined in the on-board multiplexer with any manual inputs 
before being sent to the helicopter servos. 

All information received by the 486, including information received from the GPS re- 
ceivers and all information sent to the manual controller is logged throughout each flight. 
The data is sent down to the ground station via wireless Ethernet and stored there on hard 
disk. Any measurement made during the flight can be reproduced in the lab for system 
debugging, system identification, and control law development. 

5 Control Laws 

One control law has been developed - a robust hover control. The ARL’s approach to control 
development has been largely experimental. No cross couplings between the yaw, vertical, 
lateral, or longitudinal dynamics have been modeled. 

The yaw dynamics are controlled using only heading information. The heading rate 
information is ignored as the existing gyroscope provides negative feedback of yaw rate at a 
higher update with less delay than possible with GPS yaw rate information. 

Altitude is controlled by feeding back both altitude and altitude rate (PD feedback). 

The lateral and longitudinal dynamics require a more complex control strategy. Design 
was based on successive loop closure, where an inner “attitude loop” was first closed, followed 
by closure of outer “position” loops. 

The first level of feedback is provided by the Hiller paddles, inherent in the helicopter’s 
basic design. Since the paddles provide attitude rate feedback, the GPS attitude rate signals 
can be ignored. The second loop closure feeds back roll and pitch information. The outer 
loops feed back vehicle position and velocity. 

6 Navigation 

The competition objectives can be achieved with four types of trajectories - search, travel, 
retrieval, and inspection. Commands to switch trajectory modes will be made autonomously. 

Due to the structured nature of the competition field, the GPS coordinates of the initial 
search trajectory is known a priori. This search will be conducted in a back-and-forth, lawn 
mower-like pattern. The search will end once the entire field has been surveyed or all five 
barrels have been definitively located. 

As a result of the relatively short trajectory lengths, travel can be completed with a 
quasi-static control system about hover. The helicopter can be made to follow trajectories 
by quantizing the trajectory and commanding the helicopter to hover at each successive 
point along the path. 
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he relative location of the disk. The retrieval trajectory will ensure that the helicopter does 
not exceed position or velocity limits during disk acquisition. d 

If conditions at the contest site require that pictures of the barrel emblems be taken from 

f h efT nt altltude than that used ^ normal flight, the inspection procedure will command 
the helicopter to hover over a barrel and slowly descend or ascend to this new altitude \fter 

a sufficient amount of time to insure that a quality image has been taken, the helicopter will 
slowly resume its original altitude. nencopter will 


7 Drum Identification 

There is some known information about the drums, including: 

• There are five 55-gallon black plastic drums. 

‘ arena boundary. ““ b °" ndaneS * “ d are “ <*“ ^ **« -ters «„ any 

■ to be either my exposed ° r partiaiiy buried ’ either staiidin s 

• The label will be visible from directly above. 

• One of the barrels will have a ferromagnetic disk resting on it. 

Two on-board video cameras pointing straight down provide stereo range information which 
is used to find these drums. Because the field is effectively flat, any large incre^ in the teal 
ground level of the helicopter can be assumed to be a barrel. A wide camera angle provides 
a arge stereo field, allowing filtering processes to find barrels not only directly beneath the 

of the'ir , /h t °u lt v fr ° nt “ d Sid6S ' When cou P led with th « accurate GPS knowledge 
the location of the helicopter when each image was taken, a complete map of the field can 

be constructed that locates the barrels to within one meter. Overlapping video images sewe 

eliminate noise and faulty readings by looking at each point from several different angles. 

8 Disk Detection and Retrieval System 

Because the exact location of the ferromagnetic disk is not known, an adaptive system was 
developed to pick out the disk from its surroundings and retrieve it. The video image 
from one of the cameras is analyzed using a variety of color processes. The first algorithm 
breaks down the image into its red, blue, and intensity components. Because the disk is th“ 

retiTned A , C ° mp0, * ent °" th f com P etitio “ held, only the red information is 

tamed A Gaussian filter then suppresses almost all the noise resulting from the grass and 

he wire, ess transmission. Finally, a threshold filter produces a final blL-and-wfifteTmage 

can then h‘ re y 1 eXCeP * r' ' he diSk ' The ceatroid of the disk in the ™ia field of view 
can then be easily ascertained. 
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Using the accurate position and attitude information provided by GPS, the location of 
the disk is computed and then compared to the determined locations of the barrels If a 
match is made, control of helicopter position is given to the vision system, which attempts 
o place the disk in the center of its image (the approximate location of the disk retriever 
magnets.) The retriever, made up of four permanent magnets mounted on a wooden frame, 
needs to be within one inch of the disk to attract and grab it. Once the disk is secured the 
helicopter proceeds to the starting area for landing. 


9 Conclusions 

This research is a continuation of previous work which demonstrated the first fully au 
tonomous control of an unstable air vehicle using GPS as the only sensor. The main ad- 
vancement of this work is the incorporation of real-time vision into the planning of the 
vehicle path and the use of video within the position-controlling loop to locate and interact 
with an object. In the future, the ARL helicopter will continue to serve as a testbed for 
integrating GPS with other sensing and control technologies. As the mass, size, and power 
consumption of GPS receivers is reduced, it is expected that GPS will become an important 
sensing technology for future autonomous systems. 
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1 Abstract 

Recent results in merging Carrier- Phase Differential 
GPS (CDGPS) sensing with vision-based sensing to 
control a small autonomous helicopter are presented. 
The helicopter is a heavily modified yet low-cost kit 
helicopter that includes a full on-board CDGPS sys- 
tem, stereo video cameras, and sufficient computa- 
tional capabilities to perform fully autonomous mis- 
sions. The demonstration mission presented here is 
a search of a field to find an object followed by the 
tracking of that object (using the vision system) as 
it moves around the field. The helicopter system is 
described and the performance in accomplishing the 
task is presented. 

2 Introduction 

The control of autonomous unmanned aerial vehicles 
(UAVs) has become a topic of considerable interest 
in recent years. The availability of these vehicles will 
enable tasks to be accomplished with greatly reduced 
levels of pilot workload (e.g. when teleoperation is 
used) including complete autonomy in the execution 
of a mission. Possible applications include remote sur- 
veying and aerial mapping, power line inspection, crop 
dusting, fire fighting, and movie filming. In order to 
accomplish these missions, however, progress needs to 
be made in the areas of navigation and position con- 
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trol, attitude sensing and stabilization, sensing of the 
surrounding environment, and autonomous task exe- 
cution strategies. 

Since 1995 the Stanford Aerospace Robotics Lab- 
oratory (ARL) has been operating a small, fully au- 
tonomous helicopter, HUMMINGBIRD, as part of a 
program of basic research to develop these basic ca- 
pabilities for autonomous operation of UAVs. Previ- 
ous work in this program focused on demonstrating 
the feasibility of using Carrier- Phase Differential GPS 
(CDGPS) as a sensor for attitude and position con- 
trol as well as for navigation. Losing this system alone 
(i.e. as the only sensor on board the helicopter), an 
autonomous control system was implemented that sta- 
bilized and controlled the helicopter with performance 
matching or exceeding that of an expert human pilot. 
Precision flight was experimentally demonstrated by 
performing autonomous hover, automatic retrieval of 
a ferromagnetic disk using a magnet-tether manipula- 
tor, and autonomous landing tasks [1], [2], 

While significant, the autonomous capabilities of 
the helicopter demonstrated previously were limited 
to missions that could be expressed entirely in a GPS 
frame. That is, commands to fly trajectories, retrieve 
objects, or land were all possible only if the location 
of the paths, points or objects could be expressed as 
GPS coordinates (known apriori). Many tasks of cur- 
rent and future interest require, however, that the he- 
licopter be able to sense the presence of objects-of- 
interest in the environment whose GPS coordinates 
are unknown. An obvious example is a search-task in 
which the objective is to find and then track or retrieve 
an object. Accomplishing these types of tasks requires 
that the helicopter also be equipped with sensing sys- 
tems that can detect the presence of the objects-of- 
interest. 

Described below are the results of current research 
activities in the Stanford ARL that have addressed 
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these issues. This research has involved the develop- 
ment of a small autonomous helicopter, HUMMING- 
BIRD, and the demonstration of its use in performing 
tasks that combine both vision and GPS based control 
modes. 

3 Related Research 

Several other researchers have explored the develop- 
ment of autonomous helicopters. The earliest success- 
ful work was in the early 1990’s by Michio Sugeno at 
the Tokyo Institute of Technology [3] using a Yamaha 
R-50 helicopter. The TIT helicopter used gyroscopes, 
accelerometers, and a laser altimeter as sensors and 
implemented a fuzzy logic controller. The helicopter 
was able to use image guidance to find a landing spot 
and land on the stationary target. More recent work 
includes that of Stanford, MIT and CMU all of whom 
have won the Association for Unmanned Vehicle Sys- 
tems International aerial robotics competition. Stan- 
ford won in 1995 using a precursor to the HUMMING- 
BIRD helicopter described in this paper that flew a 
predetermined trajectory and retrieved an object suc- 
cessfully from the ground. This helicopter demon- 
strated the feasibility of using CDGPS as the only 
sensor for both attitude stabilization and navigation. 
In 1996, a team from MIT, Boston University, and 
Draper Labs [4] won with a helicopter that used an in- 
ertial navigation unit for attitude and Carrier-Phase 
Differential GPS (CDGPS) for position. This MIT 
helicopter carried a camera onboard and was able to 
map the location of objects on the ground. Finally, 
Carnegie Mellon University won in 1997 flying a Yamaha 
R-50 [5] that mapped and identified ground targets. 

In each of these cases, the ground target was station- 
ary throughout the helicopter flight. 

4 Helicopter Testbed 

The HUMMINGBIRD helicopter used for these exper- 
iments is a heavily modified Schluter Futura model he- 
licopter (Figure 1). It was selected for its low empty 
weight, its large payload capacity (over 25 pounds), 
and its 15 minute endurance. The modifications in- 
cluded stiffening of the structure to accommodate the 
mounting of the video and electronic components, and 
the replacement of its standard two- horsepower sin- 
gle cylinder engine by a five- horse power dual-cylinder 
engine intended for use on model airplanes. This re- 
quired the design and construction of a new engine 
mounting plate and modification of the power trans- 
mission devices. The Futura is equipped with two 
stabilization devices, “Hiller paddles” and a mechan- 



Figure 1: HUMMINGBIRD Helicopter 

ical rate gyro. The “Hiller paddles” are used to slow 
the lateral and longitudinal dynamics. The rate gyro 
is used in a simple electrical feedback loop to slow 
the yaw dynamics. Both of these devices are standard 
equipment for helicopters of this size, and are essential 
to permit manual flight operation. 

The computing and sensing components onboard 
the HUMMINGBIRD vehicle include two GPS receivers, 

4 GPS antennae, two video cameras, a wireless video 
link, a wireless ethernet link, a 9600 baud com link, a 
486 class computer and two HCI1 computers. The 
supporting ground station includes a GPS antenna 
and receiver and a dual-Pentium computer (for the 
vision data processing). 

5 GPS Sensing System 

The HUMMINGBIRD helicopter uses the Global Po- 
sitioning System (GPS) as its primary sensor, both 
for navigation and stabilization. This is a satellite- 
based navigation system which offers absolute posi- 
tioning in a world-fixed reference frame anywhere on 
the globe. GPS offers many additional advantages for 
an airborne navigation system, including (1) integra- 
tion of all sensors into a single unit; (2) drift free rate 
information; (3) no moving parts; and (4) relatively 
small size and power consumption. 

The GPS technology employed by the ARL is called 
Carrier-Phase Differential GPS (CDGPS) since it uses 
the carrier signal rather than the GPS code to com- 
pute position. This carrier wave has a wavelength of 
19 cm and can be tracked very precisely. The HUM- 
MINGBIRD helicopter employs an algorithm which 
examines the raw phase measurements taken at two 
non-collocated receivers and computes the relative (dif- 
ferential) position between them in a world-fixed ref- 
erence frame to an accuracy of 2-3 cm. 



The GPS receivers used on HUMMINGBIRD are 
Trimble TANS Quadrex units. These receivers are 
well suited to attitude determination, since they can 
take in multiplexed signals from up to four antennas. 
In addition the TANS outputs raw phase measure- 
ments at 10 Hz, which is fast enough for the stabiliza- 
tion of the helicopter dynamics. 

5.1 Attitude Sensing 

The GPS attitude sensor consists of a single on-board 
GPS receiver and an array of four active GPS anten- 
nas. These antennas are arrayed in an upward- facing 
diamond configuration just below the main rotor disk: 
one (master) antenna mounted on the tail boom, one 
mounted on the fuselage forward of the main rotor 
mast, and two lateral antennas mounted on towers 
attached to the electronics payload plate. This place- 
ment is intended to minimize occlusion of the satellites 
by the helicopter airframe. Since the rotor blades are 
constructed of wood, have a small cross section (6 cm) 
when compared to the wavelength being considered 
(19 cm), and are spinning rapidly, there is little in- 
terference as a result of the antennas being below the 
rotor disk. 

GPS signals from the four antennas are multiplexed 
into the receiver. The receiver takes carrier phase 
measurements of these signals and outputs these re- 
sults at 10 Hz to the on-board computer through a 
38400 baud serial communications link. The computer 
then computes the difference in position in world- fixed 
coordinates between all four antennas on the heli- 
copter. Since CDGPS difference measurements are 
accurate to 2-3 cm, and the baselines between anten- 
nas are about 1 meter, these algorithms can compute 
the helicopter attitude to an accuracy of 1-2 degrees. 
Attitude rate measurements are accurate to about 1 
degree per second. 

5.2 Position Sensing 

The GPS position sensor consists of two parts. The 
on-board section includes a second GPS receiver and 
the master antenna. This receiver computes GPS 
phase measurements and sends them directly to the 
on-board computer. The second section is the GPS 
ground reference station, which is composed of a fifth 
GPS antenna located on the ground (usually near the 
take-off point) and a third GPS receiver. GPS carrier- 
phase measurements and timing information is sent 
from this GPS ground station to the helicopter com- 
puter at 10 Hz via a 9600 baud wireless 461 MHz mo- 
dem with an RS-232 interface. The helicopter then 
computes the differential position between the ground 


station antenna and the master antenna on the tail 
boom, giving a relative position measurement accu- 
rate to 2-3 cm. Velocity measurements are accurate 
to better than 10 centimeters per second. 

In order to make this position sensor robust to 
failures, several steps were taken. First, the differ- 
ential GPS ground reference station is independent 
of the computer performing the vision processing, so 
a computer failure on the ground will not affect the 
GPS navigation. Second, all processing of GPS data 
is done on-board the helicopter, minimizing the need 
to transfer data over wireless links. Third, the posi- 
tion and attitude algorithms are independent of each 
other. If the GPS ground station and the helicopter 
were to lose contact, the helicopter would still be at- 
titude stabilized and is thus easily recoverable. 

6 Vision System 

The HUMMINGBIRD vision system consists of a pair 
of downward pointing Sony XC-999 color cameras mounted 
on the front of the helicopter. Due to the weight con- 
straints of the vehicle, the vision processing is done on 
an off-board ground station computer. Two wireless 
video transmitter units are used to send the color im- 
ages from the helicopter to a dual Pentium computer. 
After processing, the information is telemetered back 
to the on-board computer via a wireless Ethernet link. 

The vision processing is performed by a Teleos 
Corp. (now a division of Autodesk) Advanced Vi- 
sion Platform (AVP) system. This system is capable 
of performing YUV color segmentation and signum- 
of-Laplacian-of- Gaussian (SLoG) filtering and corre- 
lation at 30 Hz. The current system uses the color 
segmentation ability of AVP to identify objects. The 
segmented UV images correspond closely to red and 
blue color components and test objects of these colors 
are used in order to simplify processing. The color 
segmented images are thresholded, and then objects 
are defined (as regions of a specified size). The opti- 
mal threshold values vary due to the current lighting 
conditions. They are selected by a human operator, 
not the vision system, and can be adjusted in real 
time. The AVP system has been used by a related 
project to perform texture-based mosaicking and ob- 
ject tracking and their algorithms are being brought 
into this project [6]. 

The color based object identification is performed 
on both the left and right camera images. The corre- 
spondence problem is solved by using objects of unique 
color. By only looking at a single red and a single blue 
object, it is straightforward to locate the projection of 
each object in the two different images. With the co- 
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Figure 2: Inner Loop 



Figure 3: Outer Loop 

ordinates of the object in both images, basic stereo 
triangulation is used to calculate the location of the 
object relative to the fixed camera reference frame. 
This information is then sent through the Ethernet 
link to the on-board flight computer, which transforms 
the object location into the proper helicopter reference 
frame. This system has an accuracy of approximately 
5 cm in range and 2 cm in location at a range of 2 m. 
The object tracking algorithm runs at 10 Hz. 

7 Control Logic 

Two control loops comprise the control system for 
the HUMMINGBIRD helicopter (Figures 2 and 3) A 
high- bandwidth inner loop provides attitude stabiliza- 
tion and vehicle position control based entirely on the 
CDGPS system. A low-bandwidth outer loop provides 
position commands to the inner loop based on the vi- 
sion data. In its current form, the merger of these two 
loops is accomplished by converting the locations of 
objects- of- interest observed by the vision system (i.e. 
relative to the helicopter) into global GPS coordinates 
that can be processed by the inner loop. 

The function of “Controller I” indicated in Fig- 
ure 2 is to regulate the helicopter attitude. It gener- 
ates effective aileron, elevator and rudder commands 
in response to errors in the vehicle trim condition. 
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The function of “Controller 2” indicated in Fig- 
ure 2 is to regulate the helicopter position. It gen- 
erates effective aileron, elevator, rudder and collective 
commands in response to errors in where the vehicle is 
compared with the position commands generated by 
the Trajectory Generator. That is 
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where x , y and z are expressed in body coordinates. 
In these controllers, K\(s) and /^(-s) represent a ma- 
trix of controller transfer functions. (Note that rate 
information is provided by the CDGPS system.) 

The Trajectory Generator shown in Figure 2 cre- 
ates position time history commands for the inner loop 
to track. It has two basic modes of operation: a GPS- 
based mode and a Vision- based mode. In GPS-based 
mode, the Trajectory Generator’s function is to cre- 
ate a time history of (x, y, z) the vehicle is expected to 
fly. Typically, this has been done by preprogramming 
a series of way-points connected in time by constant 
velocity trajectories (other trajectories, e.g. optimal, 
could be implemented as well). An example appli- 
cation of this mode is the programming of a prede- 
fined search pattern in GPS coordinates. The second, 
Vision-based, mode is used when objects or locations 
are being tracked using the vision system. In this 
mode, the output of the Trajectory Generator is sim- 
ply the error signal generated by the “Vision System’ 1 
shown in the Outer Loop block diagram (Figure 3). In 
the current implementation of this mode (also shown 
in Figure 3), the relative target position error is con- 
verted into GPS coordinates by the Flight Computer. 
This allows the feedback loops active in the Inner Loop 
Control to be the same for both modes of operation. 

8 Demonstration Task 

In order to demonstrate the ability of the HLJMMING- 
BIRD helicopter to perform a task that involved the 
integration of CDGPS-based control and vision-based 
control, the task identified in Figure 4 was performed. 
It consists of three main segments. In the first, the 
vehicle is commanded to fly a “lawnmower” pattern 
over the field as it searches for the target vehicle. This 
is indicated as the dashed line in the figure between 
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Figure 4: The Demonstration Task 

points 1 and 2. During this first phase of the flight, 
the vision system is on but is not part of the con- 
trol system. Rather, if the target vehicle is detected 
(the vehicle has an orange target on its back for this 
test), its position is computed in GPS coordinates and 
saved. At the end of the search phase, the helicopter 
enters the second phase where it is commanded to fly 
from point 2 to the stored location of the target (point 
3). Once there, the control system switches modes to 
closing a vision-based tracking loop that causes the he- 
licopter to hover directly over the target vehicle. Once 
in this state, the target vehicle is then driven around 
the field and the helicopter holds station directly over 
it. Note that the control loops active in this task were 
preliminary and therefore of relatively low bandwidth 
compared to what is achievable with this system. 

The performance achieved in accomplishing this 
task is presented in Figure 5. It clearly shows the 
ability of the helicopter to track the target vehicle. 

9 Conclusions 

The ability to merge CDGPS-based control and vision- 
based control to control a small helicopter has been 
demonstrated. Further, the capabilities enabled by 
this integration have been used to demonstrate a fully 
autonomous search and tracking mission with the he- 
licopter. 
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